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ABSTRACT 

A variety of resources is needed to effectively carry 
out a comprehensive evaluation of any reasonably complex program of 
library instruction. At the State University College at Oswego, New 
York, the means and motivation for carrying out formal evaluation of 
a college library's instruction program were greatly enhanced when 
librarians cooperated with a campus research course. The library's 
program became a class project; faculty and staff shared their 
expertise; and these symbiotic efforts resulted in a fairly 
comprehensive evaluation project. The total evaluation process has 
included a faculty needs assessment, evaluation of current library 
instruction offered through freshman English and upper-level 
psychology courses , and feedback from graduating students. The 
research results have prompted improvements in the library 
instruction program, and they form a basis for continuing evaluation. 
This case study suggests that librarians look to their campus 
environments for stimulus and help, since colleges are usually rich 
in resources needed for program evaluation. (Author/KM) 
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Abstract 

The means and motivation for carrying out formal evaluation of a 
college library's Instruction program were greatly enhanced when librarians 
cooperated with a campus research course. The library's program became a 
class project; faculty and staff shared their expertise; and these symbiotic 
efforts resulted In a fairly comprehensive evaluation project. The research 
results have prompted Improvements In the library Instruction program, and 
they form a basis for continuing evaluation. This case study suggests that 
librarians look to their campus environments for stimulus and help, since 
colleges are usually rich In resources needed for program evaluation. 
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The task of obtaining useful evaluative Information for the planning, 
Implementation, and further development of library Instruction programs Is 
challenging and rewarding. This paper presents a brief case study of 
efforts to develop a reasonably comprehensive evaluation of the I Ibrary 
Instruction program at SUNY College at Oswego. In this account we outline 
how librarians, a psychology faculty member and his students, and a member 
of the student services staff used widely available resources to carry out 
the evaluation from planning through data analyses. We briefly describe 
our Instructional program and the procedures used In the various components 
of our evaluation effort, highlight salient results and how they have been 
useful thus far, and summarize the process through which this example of 
successful Intracampus cooperation evolved. Our Intent Is to encourage 
similar (perhaps better) efforts at ongoing program evaluation of library 
Instruction. 
DESCRIPTION £E COLLEGE 

The State University College at Oswego, New York Is one of 14 SUNY 
colleges of arts and science. The college, located on the shore of Lake 
Ontario, serves students from all areas of the state, Including the New York 
City area. There are approximately 8000 ful I and part-time undergraduate 
students and 600 graduate students enrolled In the college. While there Is 
a wide variety of academic programs, the most popular programs Include 
bus I ness adm In Istrat Ion 126%), education ( 12JK), soc I a I and behav lora I 
sciences (1l£), and computer science (6%). The college Is selective In Its 
admissions pol Ides and more than 25% of Its graduates enter graduate or 
profess lonal school s. 
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LIBRARY INSTRUCTION PROGRAM AI CSWEGQ 

The library Instruction program at Oswego Is Just over ten years old. 
It presently reaches close to 4000 students a year on three different 
levels. Involving thirteen librarians, 

Bas Ic Instruct Ion for freshmen Is met through Eng I Ish 1 02, a 
composition class required of almost all freshmen. During the writing of 
their research papers, students come with their classes to the library for 
two sessions dealing with use of overview articles, subject access to the 
card catalog, and use of periodical Indexes, 

Upper-level Instruction Is provided through about a hundred classes a 
year, representing seventeen departments. These classes concentrate on 
locating professional literature, using advanced library tools for the 
discipline, and knowing when to utilize Interllbrary loan and data-base 
searching. An upper-level elective library course Is the third level. 
PROCESS 

A variety of resources Is needed to effectively carry out a compre- 
hensive evaluation of any reasonably complex program of library Instruction. 
These resources typically coexist In the college environment of such In- 
structional programs. Needed are skills, knowledge, and direct Involvement 
In the areas of library Instruction, Instructional development, measurement 
and testing, research design, statistics and computation, and organizational 
change. The trick appears to be to Identify and enl 1st people who can get 
the Job done In a spirit of complementary Interdependence. 

Our evaluation project was Initiated when MIgnon Adams mentioned her 
Interest In program evaluation to Mark Morey, who was scheduled to teach a 
course In program evaluation that semester. MIgnon decided to sit In on the 
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course. Soon the development of an evaluation project for Penfleld 
Library's Instructional programs became the applied class project for the 
course. 

Everyone Involved was pleased. The students In the class (six under- 
graduates and two graduate students) were excited by the prospect of contri- 
buting to the development of a project which was clearly going to be 
continued and utilized. The library Instruction staff, notably Mlgnon Adams 
and Mary Loe, was pleased and encouraged at the prospect of assistance In 
Initiating an evaluation project. And Mark was happy to have a class 
project that fit Into his course and held promise for utilization. 

Much of the planning and development of the project took place In the 
context and assignments of the program evaluation course. As the course 
progressed the library Instruction project provided relevant examples and 
concrete tasks for the students. Everyone Involved with the class contri- 
buted In some way to every part of the process, but Individual students or 
teams of students were given primary "ownership" for each component of the 
library Instruction program. Lively discussions and true experiential 
learning resulted as class participants tried to clarify program goals, 
Identify sources of Information, and develop measurement devices and 
research procedures. 

By the end of the class a great deal of work had been done In planning 
a needs assessment, evaluation of the freshman level and our upper-level 
course, and a summary evaluation. Since that time the authors of this paper 
have been primarily responsible for the continued planning, Implementation, 
and utilization of the legacy of that Initial semester. Even now the 
various parts of the evaluation project are at different stages of develop- 
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ment and utilization, and progress continues at a steadily uneven pace. 

Each of us has played a variety of roles In this project, and while we 
have shared tasks we have also relied on our different Interests and compe- 
tencies. One example of this Is Bob Schell f s skillful manipulation of the 
Statistical Package for the Social Sciences (SPSS), an extremely useful 
computer package for the management, analysis, and reporting of statistical 
Information. Currently, responsibility for the evaluation project primarily 
belongs to Mlgnon, Mary, and the library staff. Mark and Bob still partlcl- 
pate as collaborators and consultants. 
IHE FACULTY NEEDS ASSFSSMENT 

In a needs assessment, members of key groups (such as students, faculty 
members, I Ibrarlans, and administrators) are asked to respond w Ith their 
perceptions of what needs are and how they should be met. Ideally, a needs 
assessment Is done when a program Is being established. Like most Instruc- 
tion programs, however, the one at SUNY/Oswego Just grew without ever 
receiving a formal response from Oswego faculty. 
Procedure 

In the spring of 1982, a questionnaire was developed and sent through 
campus mall with a cover letter to the 290 Oswego full-time faculty. A 
subsequent reminder was sent to department chairs two weeks later; 156 
faculty members responded. 
Measurq 

The 14-ltem questionnaire was designed to ask faculty If they felt 
students needed library Instruction, who should teach library skills, and 
whether they were familiar with the Instruction program. Faculty were asked 
to respond to each Item using a 5-polnt scale of strongly agree to strongly 
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disagree. Computer center staff coded the responses. The results were 
quantified In terms of percentages marking each answer. A factor analysis 
allowed us to Identify how Items related to one another. 
Results and Discussion 

In general, we found our faculty In strong agreement that college 
studervrs Co not enter col lege knowing how to use a I Ibrary, but that they 
definitely need that knowledge while In college. 

There was less certainty that proficiency was attained by graduation- 
less than 5% strongly agreed, with about 40% agreeing and 4056 undecided. 

Who should teach them? When similar needs assessments have been done 
on other campuses, there has sometimes been a response that only col lege 
teachers should be teaching library use, while librarians ought to be 
minding the store. Eighty-five per cent of our responding faculty Indicated 
that library Instruction was an appropriate Job for a librarian, with 13 
taking the time to write in that they felt It should be a Joint responsl- 
bl I Ity of both faculty member and I Ibrarian. We I Ike to think that this 
response was due to our active Instruction program. 

Several of the significant correlations suggested an Interesting 
conclusion. Those faculty who thought that undergraduate students needed to 
use a I Ibrary effectively tended also to be those who thought I ibrarians 
were the appropriate ones to teach them. 

The responses from the various academic departments were compared. In 
general, faculty from those departments whose students use the library most 
regularly tended to respond more positively to library Instruction. 

The questionnaire demonstrated that the faculty responding were Indeed 
supportive of the Instruction program as carried out by the librarians, and 
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that they viewed library skills as essential. Since the return of the 
questionnaires and the positive responses and comments were higher from the 
departments with which the I Ibrfiry had worked most closely, we real I zed 
again the Importance of faculty and library cooperation and the close 
relationships we have developed, 
IHE ENGLISH lflZEBQSBAM EVALUATION 

In the seventies library Instruction for freshman composition (English 
102) grew In tandem with renewed Interest In writing skills. Evaluation of 
the library sessions was done via a mix of observation, attltudlnal 
questionnaires, solicited comments from the English faculty, and a limited 
type of post-test. By 1980, when the composition course became a require- 
ment, 95% (or 66) of the sections had Incorporated the library sessions Into 
their coursework. With that degree of cooperation and the happy coincidence 
that students were randomly assigned to fall or spring sections of English 
102, we were in an excellent position to evaluate the program's Impact more 
formal ly. 
Procedure 

We designed our study to answer the following questions: 

1. Does the Eng. 102 library Instruction Increase library skills and 
knowledge? 

2. What library skills do college freshmen possess? 

3. What do students want to learn most In the Eng. 102 library 
sessions? 

4. Which student-perceived needs are we satisfying with this 
Instruction? 

The Impact of the Eng. 1 02 sess Ions on student sk I I Is was eval uated 



using a modified true experimental design* Tho fall composition students 
were given a test near the end of the semester In December after completing 
their two sessions of library Instruction. At the start of the spring 
semester the students then enrolled In Eng. 102 took the same test. Because 
of random assignments to either semester, we assumed the two groups to be 
comparable. To measure what student-perceived needs were met and unmet, the 
fall Eng. 102 students were also given an ecosystem questionnaire for rating 
their Instruction experience. 

The tested population (N = approx. 1600) consisted primarily of 
freshmen. Composition professors were asked to administer the test at the 
end of the 28 fa I I classes and at the beginning of the 38 spring classes. 
Tests given after the second week of the spring semester were disallowed. 
The resulting return was 74*, 424 from the fall and 724 from the spring. 
377 Ecosystom questionnaires were analyzed. 
Measure 

iklii Evaluation Test Ion £n^ 102 This 19-ltem multiple choice test 
was designed to measure the skills that supported the research assignment In 
Eng. 102, and It was based on the objectives of the I Ibrary presentations 
and exercises. The objectives are that the students will be able to: 

1. Use background material and other criteria to narrow topic. 

2. Know several strategies for finding appropriate subject headings, 
Including the use of LCSH. 

3. Find at least 3 subject headings that pertain to their topic. 

4. Locate and Interpret a citation from Readers' fiu[_d<v 

5. Find and use one other simple Index to periodicals, locating and 
Interpreting a citation for their topic. 
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6, Determine whether citations they find are available In Penfleld 
Library, Including a Journal's availability on microfilm, 

7. Apply what they have learned to their research paper. 

Nine additional questions on the test concerned the frequency of library use 
both !n college and In high school, and previous library Instruction, 

The Ecosystem Questionnaire Th I s quest I onn a I re, g I ven to the fall 
group, measured student satisfaction with the library presentations and 
exercises. The questions pertained to the library program Itself and how It 
helped with using specific library resources (Including librarians). For 
each of the 12 Items, students were asked to make two separate Judgments 
using a five point scale. First they were asked to Indicate how much they 
"received 11 from the program, and then how much they "desired" to receive. 
Results .and Discussion 

The data from the Skills Evaluation Test compared the library skills of 
the fall students who received Eng. 102 library Instruction with the spring 
students who had not yet had that Instruction. The fall group's mean score 
was 14.0, significantly higher (T a 21.63, P < .001) than the spring groups 
mean score of 10.6. This test measure was a conservative one, since It 
shows a group difference In spite of the fact that 55% of the spring 
students had had other courses with specific library assignments, compared 
with only 165f of the fall students. 

A discriminant analysis Indicated which skill Items were most 
Influenced by Instruction. Despite the fact that the fall students did make 
strong ga I ns In I I brary sk I I I s, they cont I nued to score poor I y on a sma I I 
number of Items. This feedback was used In making several changes: we 
clarified the confusing layout of the I lbrary f s Lfst of Periodical Holdings ; 
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we worked towards a more uniform coverage of material among all of our 
I Ibrary Instructors; and our expectations of what we can and should teach 
became more realistic. One satisfying discovery was that the two questions 
that tested for application of knowledge rather than recall were among the 
test Items which were the best discriminants. 

A discriminant analysis of the ecosystem questionnaire allowed us to 
examine survey Items for student satisfaction. The mean "desired" score was 
si Ightly higher than the mean "received" score, a common result w Ith this 
type of survey. For 1/3 of the Items, however, the differences were 
negligible (less than 0.20). The two test Items with the closest "desired" 
and "received" mean scores were, happily, the Items that stated what we 
consider to be our most Important objective: a general orientation In the 
library that helps students get started on ther research papers. These same 
Items also had the highest "desired" mean score. Indicating that new 
students are Indeed anxious to learn to use their college library. 

What students were less satisfied with was the help the sessions gave 
them locating the materials they actual ly used in writing their research 
papers. To counter unrealistic expectations, we have learned to explain our 
objectives more accurately and to emphasize the time and work Involved in 
the research process. 
PSYCHOLOGY 22G EVALUATION 

Psychology 290, Research Methods In Psychology, Is a required course at 
SUNY/Oswego for psychology majors, minors, and oncentrates. It Is 
typically taken by students as second-semester sophomore or Juniors. For 
the past seven years al I five or six sections offered each semester have 
received 2 1/2 hours of library Instruction. 
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Objectives of the library session are that students will be able to: 

1. Identify annual reviews and handbooks as compilations of research 
to use to begin on a topic. 

2. Recognize Psychological Abstracts as the most comprehensive source 
of psychological research and use It. 

3. Use citation Indexes and recognize when their use Is called for. 

4. Know when It's appropriate to have a computer search done. 

5. Apply what they've learned to their research projects. 

Four test-Items were developed from these objectives and administered 
to students In Psychology 290 Just before Instruction and also at the end of 
the semester. Means were computed for each of the Items. The test had no 
Influence on their course grades, so It was a conservative measure of their 
knowledge. 

Four sections completed both the pretest (77 respondents) and post-test 
(64 respondents). For the three questions on the pretest which related to 
the first three objectives above, there was an average per student of .2 
correct answers. In addition, there were 44 responses which used Readers 1 
Gu lit£ as an answer (the worst possible answer In each case). On the post- 
test, the number of correct answers on those three questions Increased by 
six times. Only ten students used Readers 1 Guide as an answer. 

Along with the pretest, students were also asked what they wanted 
covered In the I Ibrary cl ass session. A number of students wrote In that 
they knew nothing about the material the test covered, and that the library 
session should be concerned with this. Thus, the pretest also served to 
motivate students by signaling that the material covered In the library 
sessions would be quite 'different from that covered In their previous 
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library Instruction, 

Finally, students were asked along with the post-test for suggestions 
for future classes. There were a number of useful Ideas, Including one that 
the class (which has been taught using a search strategy model) be organized 
In the same way that the test was: given an Information problem In 
psychology, how does one go about solving It, 

While this was a simple measure using simple statistics. It gave 
valuable feedback. The pretest also helped to motivate students to learn 
new material. The value of the measure far outweighed the small amount of 
time required to create and administer It, 
THE EXIT STUDY 

Initially we hoped that we cou Id assess the long term Impact of the 
entire library Instruction program on our students. While we haven't as yet 
directly assessed the library skills, knowledge, and attitudes of our 
graduating seniors and alumni, we have had the opportunity to obtain some 
feedback from our graduating students. 
Process 

In the spring of 1982, the coordinator of program evaluation for the 
revised general education program at Oswego Initiated a four-year exit study 
which was designed to assess a random sample of our graduating seniors. The 
1964 seniors were the first to graduate under the new requirements. 
Including Engl Ish 102, Part of the measurement Included a questionnaire 
which was wr I tten to el Ic 1 1 student op I n Ions about how much the I r Oswego 
experience contributed to their academic and Intellectual development In a 
number of Important areas of general education. One of the Items on the 
questionnaire directly asked the students to evaluate their Improvement In 
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the "ability to conduct research In a typical academic library." (It Is 
very likely that a similar Item will be Included In an alumni survey Instru- 
ment* currentl y being developed at the College,) The questionnaire format 
required that the student make two separate Judgments about each of the 
statements on the form. First they were to Indicate how much Improvement 
they obtained In a given area, and then they were to Indicate how much they 
had desired to Improve In that area through their Oswego experience. Addi- 
tional ly, tho students were asked to explain their responses and make 
suggestions for Improvement If their "obtained" and "desired" Improvement 
ratings differed by two or more on the five point scale from "1...very 
I Ittle or none" to "5... a great deal," 
Results fllld Discussion 

The responses to the library Item for the first two year* 5 of the study 
(1982 and 1983) have been analyzed and suggest some Interesting preliminary 
Inferences as we await future results. As Is typical In the use of such a 
response format, the "obtained" ratings for the I Ibrary Item In both 1982 
and 1983 were lower than the "desired" ratings. However, the difference In 
1983 was considerably less than In 1982. In fact, a discriminant function 
analysis of the 1982 data (N=165) Indicated that the I Ibrary Item was one 
which best discriminated between "obtained" and "desired" responses. The 
mean "obtained" rating (3.85 with a standard error of the mean of 0^099) was 
significantly lower than the mean for the "desired" ratings (4.13 with a 
standard error of 0.08). The 1983 discriminant analysis (N=207) Indicated a 
smaller discrepancy between the "obtained" and "desired" ratings on the 
Item. The "obtained" rating was 3.94 (standard error of the mean = 0.075) 
and the "desired" mean rating was 4.11 (standard error of the mean = 0.073), 
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A separate t-test (t=-2.50, p=0.013) suggests that this 1983 discrepancy, 
though small. Is reliable. 

It will be Interesting to see If this apparent trend for the reduction 
of the disparity between "obtained" and "desired" Improvement In this skill 
area Is continued In future samples. Several students commented that they 
thought more I Ibrary Instruction should be required of students and that 
students should engage In more I Ibrary research throughout their col lege 
career. The library reaches more students with Instruction In the now- 
required freshman composition course (Eng. 102), and the expansion of Its 
other Instruction program components; this could contribute to even better 
matches between "obtained" and "desired" responses to the I Ibrary Item as 
the Exit Study continues. 
C0NC1 US I ON 

Though serendipity and circumstances In Oswego's curriculum certainly 
helped, our evaluation program relied on resources and personnel strengths 
that exist on almost every college campus. The evaluation projects provided 
Information about our library Instruction program that we used to make 
Improvements. The studies also revealed our library program's strengths, 
and they now constitute a basic foundation for continued evaluation. Our 
hope Is that this case study encourages other librarians to mobilize similar 
efforts to evaluate and Improve their Instruction programs. 



